Papers with probabilistic tokenization
Improving Consistency in LLM Inference using Probabilistic Tokenization (2025.findings-naacl)
Copied to clipboard
| Challenge: | Prior work has shown that probabilistic tokenizations can generate multiple tokenization of the same input string. |
| Approach: | They propose a method to leverage the multiple tokenization capabilities of modern LLM tokenizers. |
| Outcome: | The proposed method improves the self-consistency of large language models by generating multiple tokenizations. |